On the metric evaluation of human visual system computational models

Since the beginning of experiments on visual perception, it has been evident that it is difficult to model the behavior of the human visual system (HVS) in a simple way. Painters and poets experienced first some visual effects and they often used them for artworks. In these cases the perceptual phenomenon was described using qualitative terms and with “suggestive” and emotional words. In this paper I present the main empirical methods (without neurophysiological techniques) for metric analysis of some processes of visual perception, especially color perception. For each method I clarify the meaning, experimental conditions, advantages and disadvantages. Then I discuss in practical and philosophical terms the fact that is impossible to evaluate a HVS computational model considering the comparison between colors predicted by the model and colors perceived by the human observer. This is because we cannot compare the concept of color related to humans with the concept of color of the computational model of perception. The former cannot be directly measured because is a partially endogenous sensation of the observer, the latter is strictly metric because is related to a numerical system of color representation. The approaches that try to compare these “colors” are named interperceptual by the author because they try to compare different perception contexts. Finally it is presented a new method for the metric evaluation of computational models of color perception. This model is named “intra-perceptual” and uses a measure of color difference in a color space. 1. Different methods for the human visual system analysis Methods for a quantitative analysis of the HVS can be classified in three classes: 1. Methods that use bipartite field of view. 2. Methods that measure the response to a direct stimulus. 3. Methods that use a target and a reference patch. One example of the first group is the method used by Edwin Land for his first formulation of Retinex. This method is based on the comparison of stimuli in a visual field that is artificially bipartite. The tester asks the observer to make a perceptual match between different patches. This test can be used to gain data about visual perception. The second kind of methods is based on a direct evaluation of a stimulus presented in controlled condition. The observer is asked to answer to a yes/no question. In other cases the tester explains the observer a perceptual condition to be reached: the subject is asked to set one of more parameters of the stimulus until that perceptual condition is verified. The third class of methods is often used by psychologists. They use a reference patch (usually a gray or colored scale) and a target patch. Tester asks the observer to make a match between the target patch and a color (or gray value) on the reference patch. 2. Going out from the vacuum tube All these methods have advantages and disadvantages. Roughly speaking, the main advantage is that these methods allow gaining data to understand ‘trends’ in the HVS’s behavior, while the main disadvantage is that these tests are made in controlled conditions, and thus the data is not directly applicable to more complex contexts. To clarify this idea we can think about the experiment made by Netwon (invented by Galileo Galilei) on the measurement of the time needed for a stone and a feather to fall from the same altitude in a vacuum tube. He collected data about gravity in a controlled condition. Everybody knows that if we make the same experiment in uncontrolled conditions we have different results, and the results are much more different if the conditions are much more different from the controlled ones. Moreover it is very difficult to make predictions about the trajectory of a feather in uncontrolled condition because of the complexity of feather’s structure. Following this metaphor it is now evident that it is hazardous to use data gained from experiments results in a vacuum tube (simple and controlled context) to predict what can happen in the real world (uncontrolled condition). It is also evident that, in the real world, while we can predict the trajectory of the stone with a small error (simple HVS behavior), we cannot predict at all the trajectory of the feather (complex HVS behavior). Table 1 shows the metaphor between the Newton’s Experiment and Test and Measurements of HVS behavior. Tab. 1: Link between Newton’s Experiment and Test and Measurements of HVS behavior. Newton’s Experiment Tests and Measurement of HVS behavior Vacuum tube Simple context (controlled) Stone Main and simple HVS behavior Feather Detailed and complex HVS behavior Real world Complex context (uncontrolled) 3. Making computational models of HVS color perception, and evaluating them Even though it is difficult to model the HVS behavior in complex contexts, a lot of researchers made exciting computational models that can predict human color perception [1]. Some of them were tested also on particular patterns in order to quantify the ‘ability’ of such algorithms to predict with certain error the color appearance. The problem with this evaluation method is that they compare the color predicted by the computational model with the color perceived by the observer in a way that is not precisely defined. Moreover, in my opinion, these colors cannot be compared. This is because the concept of color for a human observer is a partially endogenous property while the color for a virtual observer (a computational model) is a numerical representation in a defined color space. 4. Human and machines: different kind of colors It is well know that representing colors in a color space is a difficult task if we ask the color space to have some properties. There are a lot of color spaces that try to exploit some properties like uniform scale, homogeneous perceptual difference and so on. In my opinion every color space that is not gamut constrained can be used to represent colors for a machine (e.g. XYZ) while it is not possible, in my opinion, to represent colors perceived by a human observer. This claim seems to be contradictory since color spaces was developed starting from the trichromatic assumption and trying to fit the colors perceivable by a standard human observer. Because of this claim it seems impossible to make a computational model (working on data representing colors) that simulates some properties of the human color perception. Assuming that exists a color space that represents colors in a numerical way for machine purposes, it still remains the problem of understanding how to use this data to make a comparison between colors perceived by a human observer and predictions made by computational models of color perception. 5. A walk in the vacuum tube, searching the exit door 5.1 Direct absolute comparison is impossible In fig. 1 it is shown a simplified possible schema of direct comparison between human and machine perceived colors. It is evident that a comparison is impossible because of different nature of these two concepts; it is like “summing apples and potatoes”. Real scene Color "C" Virtual Observer (computational model of color perception) Human Observer perception of color "C" becomes "CHO" "C" coded into a color space becomes "CVO". Can we compare CVO and CHO? NO! Fig. 1: Schema of direct absolute comparison 5.2 Relative comparison is impossible if reference is out of context Fig. 2 shows the schema of classic method to evaluate computational models of color perception. The human observer is asked to make a color match between a color “C” and a patch of a scale “Sx”. Usually the evaluation is made by comparing the XYZ coordinates of the computed color CVO with the XYZ coordinates of the patch Sx selected by the human observer. This method is ill posed because the scale S is involved in the perceptual process of the human observer but is not used as input for the virtual observer. In this way the reference scale can be considered “out of context” for the virtual observer while it is not for the human observer. Due to that it seems very important that the virtual observer can access exactly the same visual information of the human observer. In the next paragraph we will see that this is still not enough. In controlled conditions Color "C" In controlled conditions Scale "S" of known XYZ colors Virtual Observer (computational model of color perception) "C" coded in XYZ color space becomes "CVO" Human Observer match between "C" and "Sx" CVO has the same XYZ of the patch Sx? Fig. 2: Schema of relative comparison (reference scale out of context) 5.3 Relative comparison is still impossible if reference is in context Fig. 3 shows the schema with an added arrow indicating that the scale is fed into the virtual observer pipeline; this schema is not so different from the one in fig. 2. The problem of an ill posed method still remains because we do not know anything about the perception of the patch Sx by the human observer in that controlled context. At least we can make assumptions on the perception of the patch Sx of the scale “S” but this is not correct if we think that the human perception of a color can change heavily respect to a lot of parameters: ‘color’ of the light source, distribution of light, average intensity of light, spatial distribution of reflectance in the scene, apparent dimension of the patch on the retina, and so on. In controlled conditions Color "C" In controlled conditions Scale "S" of known XYZ colors Virtual Observer (computational model of color perception) "C" coded in XYZ color space becomes "CVO" Human Observer match between "C" and "Sx" CVO has the same XYZ of the patch Sx? Fig. 3: Schema of relative comparison (reference scale in context) 5.4 Intra-perceptual comparison in controlled condition Fig. 4 shows the proposed method for comparison of human and machine perceived colors. The main idea is that human and machine colors cannot be compared directly and the relative comparison of these colors is not appropriate (as previously discussed). Thus the idea is to make two different matches in two perceptual domains: the human observer domain and the virtual observer domain. We can decide to measure the computational model ability to predict the color appearance of a color “C” by asking a tester to make a match using the scale “S”. Then the computational model predicts the appearance of “C” in a color space generating “CVO”. The virtual observer also perceives the scale “S” and thus we have the values “SVO”. To evaluate the computational model we have to check if the two colors “CVO” and “SVO,x” match. This procedure is simple and it is not difficult to propose a metric in a color space to evaluate the error done by the computational model in predicting the appearance of “C”. To exit the vacuum tube we need another small step. In controlled conditions Color "C" In controlled conditions Scale "S" of colors Virtual Observer (computational model of color perception) "C" →"CVO" "S" → "SVO"